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Highlights 


Why  GAO  Did  This  Study 

The  U.S.  Customs  Service  is  in 
the  early  stages  of  a  multiyear, 
multibillion-dollar  project:  the 
Automated  Commercial 
Environment  (ACE),  a  new 
import  processing  system  that  is 
to  support  effective  and  efficient 
movement  of  goods  into  the 
United  States.  By  congressional 
mandate.  Customs’  spending 
plans  for  ACE  must  meet  certain 
conditions,  including  being 
reviewed  by  GAO.  In  this  study, 
GAO  addresses  whether 
Customs’  latest  plan  satisfies 
congressional  conditions  and  is 
consistent  with  open  GAO 
recommendations,  and  it 
identifies  opportunities  for 
strengthening  project 
management. 


What  GAO  Recommends 

To  increase  the  chances  of 
delivering  needed  system 
capabilities  on  time  and  within 
budget,  GAO  is  making 
recommendations  to  the 
commissioner  aimed  at  improving 
Customs’  management  of  ACE, 
including  strengthening  system 
alignment  with  Customs’ 
enterprise  architecture,  cost 
estimating,  human  capital 
capacity,  software  process 
maturity,  and  sequencing  of 
incremental  releases.  Customs 
concurred  with  GAO’s 
recommendations  and  described 
specific  actions  that  it  is  taking  to 
respond  to  each. 


United  States  General  Accounting  Office 


May  2002 


CUSTOMS  SERVICE  MODERNIZATION 
Management  Improvements  Needed  on 
High-Risk  Automated  Commercial 
Environment  Project 


Highlights  of  GAO-02-645,  a  report  to  the  Subcommittee  on  Treasury  and  General  Government,  Senate 
Committee  on  Appropriations,  and  the  Subcommittee  on  Treasury,  Postal  Service,  and  General 
Government.  House  Committee  on  Appropriations. 


What  GAO  Found 

Customs’  February  2002  ACE  spending  plan  is  the  second  in  a  series  of 
legislatively  required  plans.  This  plan  covers  certain  project  management 
tasks  as  well  as  the  definition,  design,  and  development  of  the  first  ACE 
increment.  GAO  found  that  the  plan  meets  the  legislative  conditions  that 
Congress  imposed  on  Customs  and  is  consistent  with  GAO’s  open 
recommendations.  Nevertheless,  investment  in  ACE  is  a  high-risk 
endeavor  for  several  reasons: 


•  The  system’s  size,  performance  parameters,  and  organizational  impact 
make  it  technically  and  managerially  complex.  For  example,  the 
estimated  cost  for  acquisition  alone  is  about  $1.5  billion,  the  system  is 
to  operate  around-the-clock,  and  it  is  to  drive  fundamental  business 
process  reform  not  only  within  Customs,  but  also  within  numerous 
other  federal  agencies  and  countless  private  sector  importers. 

•  Customs  fell  far  short  of  key  commitments  made  in  its  first  spending 
plan  because  it  severely  underestimated  costs.  This  track  record  casts 
doubt  on  Customs’  ability  to  meet  future  commitments. 

•  Despite  progress.  Customs  still  lacks  important  acquisition 
management  controls.  For  example,  it  needs  to  update  its  enterprise 
architecture  (its  agencywide  modernization  blueprint)  to  support 
system  design  and  development,  and  it  needs  to  advance  its  acquisition 
office’s  human  capital  and  software  management  processes. 

•  Customs  has  recently  decided  to  compress  its  time  frame  for  delivering 
the  system  from  5  to  4  years.  This  exacerbates  the  level  of  project  risk 
by  introducing  more  overlap  among  incremental  system  releases. 


Because  of  the  system’s  national  importance.  Customs  is  taking  a 
schedule-driven  approach  to  acquiring  ACE.  However,  without  the 
management  capacity  to  effectively  acquire  such  a  large  and  complex 
system,  particularly  in  light  of  Customs’  performance  to  date  and  the 
accelerated  acquisition  and  deployment  schedule,  this  approach  could 
backfire.  Full  system  capabilities  may  take  longer  and  cost  more  to 
acquire,  deploy,  and  make  operational,  because  the  system  delivered 
under  the  accelerated  schedule  could  require  considerable  rework. 

ACE  Is  High  Risk  for  Several  Reasons 


This  is  a  test  for  developing  highlights  for  a  GAO  report.  The  full  report,  including  GAO's  objectives,  scope,  methodology,  and  analysis  is  available 
at  www.gao.gov/cgi-bin/getrpt7GAO-02-546.  For  additional  information  about  the  report,  contact  Randolph  C.  Hite  (202-512-3439).  To  provide  comments 
on  this  test  highlights,  contact  Keith  Fultz  (202-612-3200)  or  E-mail  HighlightsTest@gao.gov. 
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United  States  General  Acconnting  Office 
Washington,  D.C.  20548 


May  13,  2002 

The  Honorable  Byron  L.  Dorgan 
Chairman 

The  Honorable  Ben  Nighthorse  Campbell 
Ranking  Minority  Member 

Subcommittee  on  Treasury  and  General  Government 
Committee  on  Appropriations 
United  States  Senate 

The  Honorable  Ernest  J.  Istook,  Jr. 

Chairman 

The  Honorable  Steny  H.  Hoyer 
Ranking  Minority  Member 
Subcommittee  on  Treasury,  Postal  Service, 
and  General  Government 
Committee  on  Appropriations 
House  of  Representatives 

Pursuant  to  the  U.S.  Customs  Service’s  fiscal  year  2002  appropriation,^ 
Customs  submitted  to  Congress  in  February  2002  its  second  expenditure 
plan,  requesting  $206.9  million  for  its  Automated  Commercial  Environment 
(ACE)  project.  ACE  is  intended  to  be  Customs’  new  import  processing 
system  and  the  first  project  under  the  Customs  Modernization  Program.  As 
required  by  the  act,  we  reviewed  the  expenditure  plan.  Our  objectives  were 
to  (1)  determine  whether  the  second  plan  satisfied  the  legislative 
conditions  specified  in  the  act,  (2)  determine  whether  the  plan  is  consistent 
with  our  open  ACE  recommendations,  and  (3)  provide  observations  about 
the  second  plan  and  Customs’  management  of  ACE. 

On  March  8  and  11,  2002,  we  briefed  your  offices  on  the  results  of  this 
review.  This  report  officially  transmits  to  you  the  results  of  our  work  and 
the  recommendations  we  made  to  the  commissioner  of  Customs.  The  full 
briefing,  including  our  scope  and  methodology,  is  reprinted  as  appendix  I. 

In  brief,  we  determined  that  Customs’  expenditure  plan  satisfies  the 
legislative  conditions  specified  in  the  appropriations  act.  The  plan  provides 
for  (1)  meeting  the  Office  of  Management  and  Budget’s  (0MB)  capital 
planning  and  investment  control  review  requirements;  (2)  complying  with 


'  Public  Law  107-67,  dated  Nov.  12,  2001. 
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Customs’  enterprise  architecture;^  and  (3)  complying  with  federal 
acquisition  rules,  requirements,  guidelines,  and  systems  acquisition 
management  practices.  Further,  it  was  reviewed  and  approved  by  Customs’ 
Investment  Review  Board,^  the  Department  of  the  Treasury,  and  0MB. 

Regarding  the  second  objective,  activities  described  in  the  plan  are 
consistent  with  our  open  recommendations  for  Customs  to  (1)  justify  and 
make  investment  decisions  incrementally;  (2)  strengthen  ACE  software 
acquisition  management;  (3)  immediately  transfer  responsibility  and 
accountability  for  the  International  Trade  Data  System^  (ITDS)  pilot  to  the 
ACE  modernization  program  manager;  (4)  include  any  plans  for  further 
investing  in  the  ITDS  pilot,  including  cost,  benefit,  and  risk  justification,  in 
the  next  ACE  expenditure  plan;  and  (5)  clarify  the  roles  and  responsibilities 
of  the  ACE  modernization  independent  verification  and  validation®  (IV&V) 
agent  to  ensure  that  independence  is  not  compromised. 

Finally,  we  made  the  following  observations: 

•  ACE  is  technically  and  managerially  complex  and  challenging.  Planned 
ACE  functional  and  performance  parameters  are  demanding,  and  the 
system  is  estimated  to  cost  at  least  $1.5  billion  just  to  put  in  place,  not 
including  operation  and  maintenance.  Also,  ACE  aims  to  fundamentally 
change  Customs’  and  many  other  organizations’  business  processes 
through  the  introduction  of  new  system  capabilities.  Further,  the  system 
is  to  be  available  around-the-clock  to  provide  critical  information  and 
support  important  commercial  and  enforcement  systems. 

•  Customs  did  not  meet  key  commitments  made  in  its  first  ACE 
expenditure  plan  because  of  a  significant  underestimation  of  funding 
requirements  (actual  requirements  were  about  90  percent  higher  than 


^  An  enterprise  architecture  is  an  institutional  blueprint  for  guiding  and  constraining 
investments  in  business  process  change  and  systems. 

^  Customs’  Investment  Review  Board  makes  information  technology  funding  decisions  on 
the  basis  of  comparisons  and  trade-offs  among  competing  project  proposals. 

^  ITDS  is  a  component  of  the  ACE  project  that  is  to  support  govemmentwide  collection,  use, 
and  dissemination  of  trade  data. 

^  The  purpose  of  IV&V  is  to  provide  an  independent  review  of  system  processes  and 
products.  The  use  of  IV&V  is  a  recognized  best  practice  for  large  and  complex  systems 
development  and  acquisition  projects.  To  be  effective,  IV&V  should  be  performed  by  an 
entity  that  is  independent  of  the  processes  and  products  that  are  being  reviewed. 
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estimated).  This  history  casts  uncertainty  on  Customs’  ability  to  reliably 
estimate  costs  and  meet  future  commitments. 

•  Despite  progress,  Customs  does  not  yet  have  important  ACE 
management  controls  in  place  in  the  following  four  areas: 

1.  Enterprise  architecture:  Customs’  enterprise  architecture  has  not  yet 
been  updated  and  extended  to  support  ACE  engineering  tasks; 
consequently,  Customs  risks  having  an  enterprise  architecture  that  is 
not  sufficiently  complete  to  support  ACE  design  activities.  As  a  result, 
near-term  system  design  and  development  decisions  could  later 
necessitate  system  rework  to  align  ACE  with  the  agency’s  operating 
vision,  embodied  in  the  updated  enterprise  architecture.  If  such  rework 
is  necessary,  promised  system  capabilities  are  unlikely  to  be  delivered 
on  time  and  within  budget. 

2.  Human  capital:  ACE  is  being  managed  by  the  Customs  Modernization 
Office  (CMO).  However,  this  office  does  not  have  the  people  in  place  to 
perform  critical  system  acquisition  functions.  Moreover,  the  CMO  does 
not  have  an  effective  strategy  for  meeting  its  human  capital  needs.  This 
also  increases  the  risk  that  promised  system  capabilities  will  not  be 
delivered  on  time  and  within  budget. 

3.  Software  acquisition  management:  Customs  has  yet  to  establish 
software  acquisition  process  controls  that  are  recognized  as  best 
management  practices.®  In  particular.  Customs  has  not  begun  to 
establish  process  controls  for  determining  whether  acquired  software 
products  and  services  satisfy  contract  requirements  before  acceptance, 
nor  to  establish  related  controls  for  effective  and  efficient  transfer  of 
acquired  software  products  to  the  support  organization  responsible  for 
maintenance.  These  control  weaknesses  further  reduce  the  level  of 
assurance  that  ACE  capabilities  will  be  delivered  on  time  and  within 
budget. 


®  These  best  practices  are  described  in  the  Software  Acquisition  Capability  Maturity  Model 
(SA-CMM),  developed  by  the  Software  Engineering  Institute  (SEI).  In  this  model,  SEl  has 
provided  criteria  for  characterizing  an  organization’s  software  development  and  acquisition 
processes  according  to  five  levels  of  maturity,  with  level  2  providing  the  minimum  level  of 
acceptable  effectiveness.  The  Capability  Maturity  Model  is  a  service  mark  of  Ccimegie 
Mellon  University,  and  CMM  is  registered  in  the  U.S.  Patent  and  Trademark  Office. 
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4.  Cost  estimation:  Customs  has  not  validated  its  ACE  expenditure  plan 
cost  estimates,  and  its  estimates  of  management  reserve  costs  are  not 
justified.  As  a  result,  its  plan  does  not  provide  Congress  with  credible 
cost  information  needed  for  overseeing  the  project. 

•  Customs  has  compressed  its  ACE  acquisition  plans  from  5  to  4  years. 
This  compression  increases  the  degree  of  overlap  of  program 
increments,  which  in  turn  further  increases  the  risk  that  ACE 
capabilities  will  not  be  delivered  on  time  and  within  budget. 


Recommendations 
Executive  Action 


for 


To  improve  Customs’  ACE  modernization  management,  we  recommend 
that  the  Customs  Service  commissioner  direct  the  chief  information  officer, 
as  the  designated  modernization  executive,  to  take  the  following  actions: 


•  Before  building  each  ACE  release  (i.e.,  beginning  detailed  design  and 
development),  certify  to  Customs’  House  and  Senate  appropriations 
subcommittees  that  the  enterprise  architecture  has  been  sufficiently 
extended  to  provide  the  requisite  enterprise  design  content  and  has 
been  updated  to  ensure  consistency  and  integration  across  business 
areas. 

•  Immediately  develop  and  implement  a  CMO  human  capital  management 
strategy  that  provides  both  near-  and  long-term  solutions  to  the  CMO’s 
human  capital  capacity  limitations,  including  defining  the  office’s  skill 
and  capability  needs  in  terms  that  will  allow  Customs  to  attract  qualified 
individuals  and  that  will  provide  sufficient  rewards  and  training,  linked 
to  performance,  to  promote  their  retention. 

•  Develop  and  implement  process  controls  for  the  SEI  SA-CMM  level  2 
key  process  areas  and  the  level  3  acquisition  risk  management  key 
process  area. 

•  By  September  30,  2002,  conduct  and  report  to  Customs’  House  and 
Senate  appropriations  subcommittees  on  the  results  of  an  internal 
assessment  of  the  CMO’s  maturity  against  the  SEI  SA-CMM  level  2  key 
process  areas  and  the  level  3  acquisition  risk  management  key  process 
area. 

•  Develop  and  implement  a  rigorous  and  analytically  verifiable  cost 
estimating  program  that  embodies  the  tenets  of  effective  estimating  as 
defined  in  SETs  institutional  and  project-specific  estimating  models. 
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•  Limit  future  expenditure  plan  requests  for  management  reserve  funds  to 
10  percent  of  the  total  funds  requested  for  the  program  or  adequately 
justify  any  management  reserve  requests  in  excess  of  10  percent. 

•  Address  the  risks  associated  with  the  accelerated  ACE  acquisition 
strategy,  including  (1)  immediately  analyzing  the  risks  associated  with 
the  degree  of  design,  development,  and  testing  concurrency  across  ACE 
increments  that  is  inherent  in  Customs’  4-year,  schedule-driven 
acquisition  strategy;  (2)  reconsidering  the  merits  of  this  accelerated 
strategy;  and  (3)  within  90  days  of  the  date  of  this  briefing,  reporting  to 
Customs’  House  and  Senate  appropriations  subcommittees  on  the 
agency’s  strategy  for  going  forward  and  its  plans  for  mitigating  the 
inherent  risks  associated  with  this  strategy. 


Agency  Comments 


In  written  comments  on  a  draft  of  this  report,  the  director.  Office  of 
Planning,  U.S.  Customs  Service,  concurred  with  our  recommendations  and 
described  specific  actions  that  are  being  taken  to  respond  to  each.  The 
director’s  comments  are  reprinted  in  appendix  11. 


We  are  sending  copies  of  this  report  to  the  chairmen  and  ranking  minority 
members  of  other  Senate  and  House  committees  and  subcommittees  that 
have  authorization  and  oversight  responsibilities  for  the  Customs  Service. 
We  are  also  sending  copies  to  the  secretary  of  the  treasury,  the 
commissioner  of  the  Customs  Service,  and  the  director  of  0MB. 

Should  you  or  your  staff  have  any  questions  on  matters  discussed  in  this 
report,  please  contact  me  at  (202)  512-3439. 1  can  also  be  reached  by  e-mail 
at  HiteR@gao.gov.  Key  contributors  to  this  report  were  Mark  Bird,  Barbara 
Collier,  Tamra  Goldstein,  Randolph  Tekeley,  Scott  Pettis,  and  Aaron 
Thome. 


Randolph  C.  Hite 

Director,  Information  Technology  Architecture 
and  Systems  Issues 


Page  5 


GAO-02-545  Customs  Service  Modernization 


Appendix  I 


Customs’  Second  Automated  Commercial 
Environment  (ACE)  Expenditure  Plan 
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Customs’  Second  Automated  Commercial 
Environment  (ACE)  Expenditure  Plan 

Briefing  for  the  staffs  of  the 
Subcommittee  on  Treasury  and  General  Government 
Senate  Committee  on  Appropriations 

and 

Subcommittee  on  Treasury,  Postal  Service,  and  General  Government 
House  Committee  on  Appropriations 

March  8  and  11,  2002 
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Appendix  I 

Customs’  Second  Automated  Commercial 
Environment  (ACE)  Expenditure  Plan 
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•  Introduction 

•  Objectives,  Scope,  and  Methodology 

•  Results  in  Brief 

•  Background 

•  Results 

•  Conclusions 

•  Recommendations  for  Executive  Action 

•  Agency  Comments 


Briefing  Overview 
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Appendix  I 

Customs’  Second  Automated  Commercial 
Environment  (ACE)  Expenditure  Plan 
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Introduction 


The  U.S.  Customs  Service  is  acquiring  a  new  trade  processing  system  known  as 
the  Automated  Commercial  Environment  (ACE)/International  Trade  Data 
System  (ITDS).^  This  project  is  intended  to  fundamentally  transform  not  only 
Customs’  internal  import  processing  operations,  but  also  the  manner  in  which 
it  and  numerous  other  federal  agencies  interact  with  each  other  and  with  the 
trade  community.  According  to  Customs,  the  project  will,  for  example, 

•  enable  more  efficient  import  processing; 

•  minimize  the  gap  between  revenue  that  should  be  collected  and  revenue  that 
is  actually  collected; 

•  increase  the  flow  of  legitimate  commerce;  and 

•  provide  the  trade  community  a  single  interface  with  104  federal  agencies  that 
have  trade-related  missions. 


‘  Hereafter  referred  to  as  ACE. 
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Appendix  I 

Customs’  Second  Automated  Commercial 
Environment  (ACE)  Expenditure  Plan 
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Introduction  (cont.) 


Customs’  FY  2001  and  FY  2002  appropriations  acts^  provided  $130  million  and 
$300  million,  respectively,  for  ACE.  The  acts  specified  that  these  funds  not  be 
obligated  until  Customs  submits  to  the  Congress  for  approval  ACE  expenditure 
plans  that 

•  meet  the  Office  of  Management  and  Budget’s  (0MB)  capital  planning  and 
investment  control  review  requirements; 

•  comply  with  the  Customs  Service’s  enterprise  architecture; 

•  comply  with  federal  acquisition  rules,  requirements,  guidelines,  and  systems 
acquisition  management  practices; 

•  are  reviewed  and  approved  by  the  Customs  Investment  Review  Board,  the 
Department  of  the  Treasury,  and  0MB;  and 

•  are  reviewed  by  GAO. 


^  Public  Laws  106-554  and  107-67,  respectively. 
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Appendix  I 

Customs’  Second  Automated  Commercial 
Environment  (ACE)  Expenditure  Plan 
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Introduction  (cont.) 


On  February  1,  2002,  Customs  submitted  its  second  ACE  expenditure  plan  to 
its  House  and  Senate  appropriations  subcommittees,  seeking  release  of  $206.9 
million  from  the  ACE  appropriations. 
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Appendix  I 

Customs’  Second  Automated  Commercial 
Environment  (ACE)  Expenditure  Plan 
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Objectives,  Scope  and  Methodology 


Objectives 

As  agreed,  our  objectives  were  to 

•  determine  whether  the  second  ACE  expenditure  plan  satisfies  the  legislative 
conditions, 

•  determine  whether  the  plan  is  consistent  with  our  open  ACE 
recommendations,  and 

•  provide  observations  about  the  plan  and  Customs  management  of  ACE. 
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Appendix  I 

Customs’  Second  Automated  Commercial 
Environment  (ACE)  Expenditure  Plan 
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Objectives,  Scope  and  Methodology  (cont.) 


Scope  and  Methodology 

To  accomplish  our  objectives,  we 

•  analyzed  the  plan  against  the  legislative  conditions,  to  determine  if  the 
conditions  were  met; 

•  analyzed  the  plan  to  determine  consistency  of  actions  taken  by  Customs  in 
response  to  our  open  recommendations; 

•  interviewed  ACE  and  ITDS  program  management,  support  contractor,  and 
modernization  contractor  officials  to  understand  the  scope  and  content  of 
plans,  and  to  clarify  information  in  supporting  documentation; 

•  analyzed  ACE  supporting  documentation  to  ensure  consistency  with 
information  in  the  expenditure  plan  and  to  determine  progress  made  in 
implementing  program  management  capabilities  (e.g.,  acquisition 
management,  enterprise  architecture,  human  capital,  and  cost  and  schedule 
estimating); 
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Objectives,  Scope  and  Methodology  (cont.) 


•  determined  progress  made  against  commitments  in  first  expenditure  plan; 

•  coordinated  with  the  Treasury  inspector  general  to  avoid  duplication  of 
effort  in  reviewing  ACE; 

•  did  not  evaluate  the  ACE  Investment  Strategy  and  Cost  Benefit  Analysis, 
which  we  received  on  February  13,  2002;  and 

•  as  agreed,  did  not  independently  validate  Customs’  provided  information, 
including  cost  estimates  and  project  content  and  status  information. 

We  performed  our  work  from  December  2001  through  March  2002  in 

accordance  with  generally  accepted  government  auditing  standards. 
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Results  in  Brief:  Objective  1 


Customs’  expenditure  plan  satisfies  the  legislative  conditions. 

Legislative  conditions 

Satisfies  Does  not  satisfy 

1.  Meets  0MB  capital  planning  and  investment  control  review 
requirements,  including  0MB  Circular  A-11,  part  3.^ 

2.  Complies  with  the  Customs  Service’s  enterprise  architecture. 

3.  Complies  with  federal  acquisition  rules,  requirements,  guidelines,  and 
systems  acquisition  management  practices. 

4.  Is  reviewed  and  approved  by  the  Customs  Investment  Review  Board, 
Treasury,  and  0MB. 

5.  Is  reviewed  by  GAO. 

“  Customs  provided  us  with  a  draft  of  the  expenditure  plan  before  it  was  approved  by  Treasury  and  0MB.  We  told  Customs 
that  the  version  provided  did  not  satisfy  0MB  Circular  A-11,  part  3,  requirements.  Customs  amended  the  plan  before 
submitting  it  to  the  House  and  Senate  Appropriations  Committees. 
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Accountability  *  Integrity  *  Reliability 

Results  in  Brief:  Objective  2 

Customs’  expenditure  plan  is  consistent  with  open  GAO 
recommendations. 

GAO  open  recommendations 

Consistent 

Complete  In  progress 

Inconsistent 

1.  Justify  and  make  investment  decisions  incrementally. 

V 

2.  Strengthen  ACE  software  acquisition  management. 

V 

3.  Immediately  transfer  responsibility  and  accoimtability 
for  the  ITDS  pilot  to  the  ACE  modernization  program 
manager. 

V 

4.  Include  any  plans  for  further  investing  in  the  ITDS  pilot, 
including  cost,  benefit,  and  risk  justification,  in  the  next 
ACE  expenditure  plan. 

V 

5.  Clarify  the  roles  and  responsibilities  of  the  ACE 
modernization  independent  verification  and  validation 
(IV&V)  agents  to  ensure  that  independence  is  not 
compromised. 

V 

Page  10  of  60 

Page  15 


GAO-02-545  Customs  Service  Modernization 


Appendix  I 

Customs’  Second  Automated  Commercial 
Environment  (ACE)  Expenditure  Plan 


i 


GAO 

Accountability  *  Integrity  *  Reliability 


Results  in  Brief:  Objective  3 


Other  observations:  Customs  faces  considerable  risk  in  acquiring 
and  implementing  ACE. 


Adds  risk 

+ 

Adds  risk 

+ 

Adds  risk 

+ 

Adds  risk 

CUMULATIVE 
RISK  IS  HIGH 
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Background 


ACE  Overview 

ACE  is  intended  to  support  Customs’  efforts  to  protect  our  nation’s  borders, 

enforce  trade  laws,  and  facilitate  the  movement  of  legitimate  trade.  For 

example,  ACE  is  to 

•  support  all  Customs  functions  associated  with  trade  compliance  (such  as 
defining,  capturing,  and  managing  regulations  for  import  and  export 
compliance;  establishing  and  managing  accounts  for  trade  participants;  and 
verifying  credentials  and  managing  compliance  levels); 

•  support  other  functions  such  as  data  analysis  for  trade  law  enforcement  and 
border  security  purposes; 

•  support  the  trade  information  collection,  processing,  and  dissemination 
needs  of  104  other  government  agencies;  and 

•  provide  the  trade  community  a  single  interface  to  the  federal  government. 
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Background  (cont.) 


According  to  Customs,  ACE  will  permit,  for  example,  more  efficient  import  and 
export  operations;  increased  legitimate  cargo  flow;  reduced  paper  burden;  and 
more  efficient  and  effective  account  management,  including  revenue  billing 
and  collection  operations. 

We  have  testified  that  the  need  for  Customs  to  modernize  its  trade  processing 
capability  is  undeniable  because,  among  other  things.  Customs’  existing  import 
processing  system  was  paper-intensive,  error-prone,  transaction-based,  and  out 
of  step  with  the  just-in-time  inventory  practices  of  the  trade  community.^ 


U.S.  General  Accounting  Office,  U.S.  Customs  Service:  Observations  on  Selected  Operations  and  Program  Issues, 
GAO/T-GGD/AIMD-00-150  (Washington,  D.C.:  Apr.  20,  2000). 
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Background  (cont.) 


ACE  Acquisition  Strategy 

Customs  is  using  an  integration  contractor,  supported  by  subcontractors,  to 
develop  and  deploy  ACE.  The  contractor  and  its  subcontractors  are 
collectively  called  the  e-Customs  Partnership  or  eCP. 

Initial  modernization  contract  activities  include  establishing  a  management 
foundation  (e.g.,  people,  processes,  tools)  for  Customs’  overall  modernization 
program.  ACE  is  to  be  the  first  modernized  system  under  this  program. 

Customs’  current  plans  call  for  release  of  ACE  capabilities  in  four  increments 
implemented  over  4  years.  Each  increment  is  to  contain  discrete  functionality, 
while  building  on  the  functionality  in  the  previous  increments. 


Page  14  of  60 


Page  19 


GAO-02-545  Customs  Service  Modernization 


Appendix  I 

Customs’  Second  Automated  Commercial 
Environment  (ACE)  Expenditure  Plan 


i 


GAO 

Accountability  *  Integrity  *  Reliability 


ACE  Management  Strnctnre 


Backgronnd  (cont.) 
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Background  (cont.) 


Chronology  of  ACE  Expenditure  Plans  and  Funding 

On  March  5  and  6,  2001,  Customs’  House  and  Senate  appropriation 
subcommittees,  respectively,  released  to  Customs  $5  million  in  stopgap 
funding  to  sustain  program  activities  until  Customs  submitted  its  first 
expenditure  plan. 

On  March  26,  2001,  Customs  submitted  its  first  ACE  expenditure  plan  to 
Congress  for  $45  million. 

On  April  23,  2001,  we  briefed  your  offices  on  the  results  of  our  review  of  the 
first  ACE  expenditure  plan.  In  summary,  we  reported  that 

•  the  plan  satisfied  the  legislative  conditions; 

•  the  plan  was  consistent  with  our  February  1999  recommendations  that 
Customs 

■  justify  and  make  ACE  investment  decisions  incrementally,  and 

■  strengthen  ACE  software  acquisition  management; 
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Background  (cont.) 


•  Customs  was  investing  in  an  ITDS  pilot  project  in  parallel  with,  and  separate 
from,  ITDS  efforts  under  the  ACE  modernization  program;  and 

•  Customs  had  defined  conflicting  roles  and  responsibilities  for  its 
independent  verification  and  validation  (IV&V)  contractor. 


Accordingly,  we  made  the  following  recommendations,  which  Customs  agreed 

to  implement: 

•  Immediately  transfer  responsibility  and  accountability  for  the  ITDS  pilot  to 
the  ACE  modernization  program  manager. 

•  Include  any  plans  for  further  investing  in  the  ITDS  pilot,  including  cost, 
benefit,  and  risk  justification,  in  the  next  ACE  expenditure  plan. 

•  Clarify  the  roles  and  responsibilities  of  the  ACE  modernization  IV&V  agent  to 
ensure  that  independence  is  not  compromised. 
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Background  (cont.) 


On  April  24  and  25,  2001,  Customs’  House  and  Senate  appropriation 
subcommittees,  respectively,  released  to  Customs  an  additional  $45  million  in 
response  to  Customs’  first  ACE  expenditure  plan. 

On  April  27,  2001,  Customs  awarded  a  5-year,  indefinite-delivery  indefinite- 
quantity  contract,  with  options  to  extend  the  contract  to  not  more  than  15 
years,  to  International  Business  Machines  (IBM)  Global  Services. 

On  February  1,  2002,  Customs  submitted  its  second  ACE  expenditure  plan, 
seeking  release  of  $206.9  million. 

The  following  two  figures  illustrate  the  recent  history  of  ACE  appropriations, 
expenditure  plans,  funding,  and  major  contract  events. 
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Background  (cont.) 


ACE  funding  releases  shown  by  fiscal  year  appropriations 


FY  2001  appropriations  =  $130  million  FY  2002  appropriations  =  $300  million 
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^^^^^^^^^Accountability  *  Integrity  *  Reliability  Background  (cont.) 


Illustrated  chronology  of  ACE  expenditure  plans  and  funding 
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Background  (cont.) 


Customs’  second  ACE  expenditure  plan  requests  funding  for  the 
following: 

Areas  of  expenditure 

Amount  (millions) 

Funds  to  cover  first  expenditure  plan  cost  variance® 

$22.4 

National  Customs  Automation  Program  (NCAP)  operations  &  maintenance 

$3.1 

Program  management  and  support 

$23.5 

SAP®  infrastructure 

$7.3 

ACE  increment  1,  release  1.0  design,  development,  test,  and  deployment 

$104.0 

ACE  increment  2,  detailed  requirements  and  planning 

$11.2 

Customs  Modernization  Office  (CMO)  management  reserve 

$35.5 

Total 

$206.9 

*  The  second  expenditure  plan  requests  funding  equivalent  to  the  difference  between  the  negotiated  price  for  the  task 
orders  ($47.4  million)  and  the  funding  released  as  a  result  of  its  first  expenditure  pian  ($25  million). 

^  These  funds  wili  be  used  to  acquire  hardware  and  supporting  infrastructure  for  Customs’  SAP  Implementation  Project,  a 
joint  project  of  the  Office  of  Finance  and  the  Office  of  Information  Technology.  SAP  AG  is  a  vendor  that  makes  a  software 
package  called  R/3,  which  is  to  be  used  by  ACE  to  account  for  payments,  collections,  and  refunds.  Additionally,  R/3  is  to 
replace  Customs’  existing  core  budget,  accounting,  procurement,  and  property  accountability  systems. 
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Background  (cont.) 

Customs  attributes  its  first  expenditure  plan  variance  to  underestimates 

or  omissions  in  the  following  areas: 

Labor:  $12.9  million  (58  percent  of  the  total  variance) 

Variance  (millions) 

Labor  rates  and  skill  mix 

$6.8 

Tasks  (e.g.,  change  management,  Stakeholder  Relations  Cormcil) 

$4.5 

Other  (e.g.,  performance-based  contracting) 

$1.6 

Other  direct  cost:  $9.9  million  (42  percent  of  the  total  variance) 

Task  order  planning 

$2.0 

Facilities  and  support  infrastructure 

$6.7 

Other  (e.g.,  performance-based  contracting,  travel  costs) 

$1.2 

Total 

$22.8 
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Results:  Objective  1 


Customs’  expenditure  plan  satisfies  the  legislative  conditions. 


Legislative  conditions  Satisfies?  Results  of  our  analysis 

1.  Plan  meets  0MB  capital  0MB  circular  A-11,  part  3  sets  requirements  for  budget 

planning  and  investment  control  justification  and  reporting  requirements  for  major 

review  requirements,  including  acquisitions.  To  respond  to  A-11  requirements, 

0MB  Circular  A-11,  part  3.  agencies  are  to  cover  key  topics  such  as: 

•  compliance  with  federal  secrurity  and  privacy 
regulations; 

•  compliance  with  Government  Paperwork 
Elimination  Act  requirements; 

•  compliance  with  strategic  planning  requirements 
that  demonstrate  how  the  project  will  use  a 
performance-based  measurement  system;  and 

•  cost  and  schedule  information. 

The  expenditure  plan  and  supporting  documentation 

address  each  of  these  key  topics. 
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Results:  Objective  1  (cont.) 

Legislative  conditions 

Satisfies? 

Results  of  our  analysis 

2.  Plan  complies  with  Customs’ 
enterprise  architecture. 

The  expenditure  plan  and  supporting  documentation 
provide  for  the  development  of  an  ACE  system 
architecture  that  is  to  comply  with  Customs’  enterprise 
architecture. 

3.  Plan  complies  with  federal 
acquisition  rules,  requirements, 
guidelines,  and  systems 
acquisition  management 
practices. 

The  expenditure  plan  and  supporting  documents 
include,  for  example,  steps  for  implementing  the 

Software  Engineering  Institute’s  (SEI)  Software 

Acquisition  Capability  Maturity  Model  (SA-CMM)^  key 
process  areas.  If  implemented  effectively,  these 
processes  should  enable  Customs  to  meet  important 
federal  acquisition  rules,  requirements,  guidelines,  and 
systems  acquisition  management  practices. 

’^Capability  Maturity  Model™  is  a  service  mark  of  Carnegie  Mellon  University,  and  CMM  is  registered  in  the  U.S.  Patent  and 
Trademark  Office.  The  SA-CMM  identifies  key  process  areas  that  are  necessary  to  effectively  manage  software-intensive 
system  acquisition.  Level  2  is  the  second  level  of  the  SA-CMM’s  five-level  scale  and  means  that  an  organization  has  the 
software  acquisition  rigor  and  discipline  to  repeat  project  successes. 
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Results:  Objective  1  (cont.) 

Legislative  conditions 

Satisfies? 

Results  of  our  analysis 

4.  Plan  is  reviewed  and  approved 
by  Customs  Investment  Review 
Board  (IRB),  Treasury,  and  0MB. 

Customs  IRB  approved  the  expenditure  plan  in  two 
parts  on  September  26,  2001,  and  October  16,  2001. 

Treasury  approved  the  plan  on  December  20,  2001. 

0MB  approved  the  plan  on  January  31,  2002. 

5.  Plan  is  reviewed  by  GAO. 

Our  review  was  completed  on  March  1,  2002. 
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Results:  Objective  2 


Customs’  expenditure  plan  is  consistent  with  GAO  open 
recommendations. 


GAO  recommendations 

Consistent? 

Actions  planned  or  taken 

Complete  In 

progress 

1.  Justify  and  make  investment 
decisions  incrementally,  and  for 
each  increment: 

Customs  expenditure  plan  and  February  12,  2002,  ACE 
Investment  Strategy  and  Cost  Benefit  Analysis*  describe 
an  incremental  approach  to  investing  in  ACE  that 

•  Use  disciplined  investment 
processes; 

•  Entails  compliance  with  Customs  investment 
management  process,  which  we  determined  meets 
OMB’s  policy  for  management  of  federal  information 
resources; 

•  Prepare  realistic  and  supportable 
benefit  expectations; 

•  Identifies  costs  and  benefits  for  the  first  ACE 
increment; 

•  Require  a  favorable  return  on 
investment  and  compliance  with 
Customs  architecture  before 
making  an  investment  decision; 
and 

•  Identifies  a  positive  return  on  investment  and 

compliance  with  Customs’  enterprise  architecture,  as 
required  by  Customs’  investment  management 
process;  and 

•  Validate  actual  costs  and  benefits 
once  an  increment  is  piloted. 

•  Plans  for  validation  of  actual  costs  and  benefits  as 
determined  in  post-implementation  reviews. 

“  We  did  not  evaluate  the  ACE  Investment  Strategy  and  Cost  Benefit  Analysis,  which  we  received  on  February  13,  2002. 
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Results:  Objective  2  (cont.) 

GAO  recommendations 

Consistent? 

Complete  In 

progress 

Actions  planned  or  taken 

2.  Strengthen  ACE  software 
acquisition  management  by 

•  establishing  an  effective 
process  improvement  program 
and  correcting  any  weaknesses 
in  ACE  software  acquisition 
processes  identified  in  this 
report  thereby  bringing  ACE 
processes  to  at  least  SEI  CMM 
level  2;  and 

•  requiring  at  least  CMM  level  2 
processes  of  all  ACE  software 
contractors 

■/ 

Customs’  expenditure  plan  includes  frmds  to 
continue  software  acquisition  process  improvement 
efforts,  including 

•  establishment  of  a  software  acquisition  process 
improvement  program,  correction  of  software 
acquisition  processes,  and  demonstration  of 
progress  toward  bringing  ACE  processes  to  at 
least  SEI  SA-CMM  level  2. 

•  Customs  instituted  a  policy  requiring  CMM  level 

2  processes  of  all  software  contractors,  and  is 
requiring  that  its  prime  modernization  contractor 
maintain  level  3  processes. 
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Results:  Objective  2  (cont.) 

GAO  recommendations 

Consistent? 

Complete  In 

progress 

Actions  planned  or  taken 

3.  Immediately  transfer 
responsibility  and  accotmtability 
for  ITDS  pilot  to  the  ACE 
modernization  program  manager. 

Effective  May  2,  2001,  Customs  transferred  all  ITDS 
program  management  responsibilities  to  the 

Customs  Modernization  Office,  which  is  headed  by 
the  ACE  modernization  program  manager. 

4.  Include  any  plans  for  further 
investing  in  the  ITDS  pilot 
(including  cost,  benefit,  and  risk 
justification)  in  the  next  ACE 
expenditme  plan. 

Customs  suspended  the  ITDS  pilot  on  September  11, 

2001,  to  devote  all  available  resources  to 
maintaining  its  “Level  One  Alert”  status  at  border 
crossings.  Customs  does  not  plan  to  restart  the  ITDS 
pilot. 

5.  Clarify  the  roles  and 
responsibilities  of  the  ACE 
modernization  IV&V  agent  to 
ensure  that  independence  is  not 
compromised. 

Customs  has  clarified  the  ACE  IV&V  agent’s  role  by 
eliminating  the  agent’s  program  support 
responsibilities  in  areas  where  it  has  IV&V 
responsibility. 
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Results:  Objective  3 


Other  observations:  Customs  faces  considerable  risk  in  acquiring 
and  implementing  ACE. 


Adds  Risk 

+ 

Adds  Risk 

+ 

Adds  Risk 

+ 

Adds  Risk 

CUMULATIVE 

RISK  IS  HIGH 
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Results:  Objective  3 


Observation  1:  ACE  is  technically  and  managerially  complex  and 

challenging. 

Planned  ACE  functional  and  performance  parameters  are  demanding.  For 

example,  ACE  is  to 

•  be  deployed  and  used  at  about  300  ports  of  entry. 

•  be  available  24  hours  a  day,  365  days  a  year.® 

•  provide  functionality  currently  provided  by  about  15  other  systems, 
including  the  following: 

■  Automated  Commercial  System  (which  tracks,  controls,  and 
processes  all  goods  imported  into  the  United  States); 

■  Automated  Export  System  (which  allows  electronic  transmissions 
of  key  export  data  from  exporters,  exporters’  agents,  and  outbound 
carriers); 


^  A  specific  system  availability  requirement  has  not  yet  been  defined. 
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Results:  Objective  3  (cont.) 


■  Treasury  Enforcement  Communications  System  (which  is  to 
support  inspection  personnel  at  borders  and  ports  of  entry  along 
with  other  mission  areas  such  as  investigations,  interdiction,  and 
intelligence  analysis);  and 

■  Seized  Currency  and  Asset  Tracking  System  (which  tracks  all 
activity  associated  with  seizures  from  the  initial  enforcement 
interest  in  the  property  until  its  final  disposition); 

•  process  about  29  million  transactions  annually; “ 

•  account  for  over  $20  billion  in  revenue  annually;  and 

•  require  the  integration  of  several  large  commercial  off-the-shelf  (COTS) 
software  products  using  customized  software  (i.e.,  glue  code).^^ 


“  According  to  Customs,  it  processed  about  29  million  import  transactions  in  1998. 

“  Customs  did  not  yet  have  software  size  and  complexity  measures.  Fftrther,  while  the  suite  of  COTS  products  to  be  used 
has  yet  to  be  defined,  Customs  plans  indicate  that  COTS  packages  will  likely  Include  customer  relationship  management 
software,  electronic  document  management  and  workflow  software,  workflow  product  and  collaborative  groupware 
software,  relational  database  management  system  (RDMS)  software,  and  online  analytical  processing  and  rules  software. 
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Results:  Objective  3  (cont.) 


ACE  is  a  large  undertaking,  as  measured  by  the  level  of  development  resources 
needed.  In  particular,  the  cost  of  developing  the  entire  system  is  estimated  to 
be  between  $1.5  and  $1.6  billion,  and  the  number  of  professional  staff  needed 
to  develop  the  system  is  estimated  to  be  about  500  to  700. 

ACE  is  also  intended  to  support  and  satisfy  the  needs  of  104  other  government 
agencies  with  international  trade-related  missions  as  well  as  the  needs  of 
thousands  of  importers.  Accordingly,  acquiring  a  system  that  meets  this  diverse 
set  of  needs  will  require  an  enormous  level  of  stakeholder  participation, 
cooperation,  and  interaction. 

ACE  is  intended  to  fundamentally  change  not  only  the  way  that  Customs 
operates  internally  but  also  the  way  that  importers  and  other  government 
agencies  operate.  The  ability  of  these  organizations  to  accept  and  absorb  such 
operational  change  is  not  guaranteed;  their  uncertainty  has  been  recognized  as 
a  reason  that  COTS-based  modernization  efforts  have  not  consistently  met 
expectations. 
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Results:  Objective  3  (cont.) 


Observation  2:  Customs  did  not  meet  key  commitments  made  in  its 

first  ACE  expenditure  plan. 

One  indication  of  future  performance,  and  thus  risk,  is  past  performance. 
Customs’  past  performance  in  acquiring  ACE  fell  short  of  the  commitments  it 
made  in  the  first  expenditure  plan. 

In  the  first  plan.  Customs  requested  $25  million  for  three  ACE  tasks: 

(1)  program  management,  (2)  enterprise  engineering,  and  (3)  requirements 
development.  Customs  reports  that  it  met  commitments  for  task  3,  but  it  did 
not  meet  all  of  the  commitments  for  tasks  1  and  2.  Specifically,  because  the 
cost  to  complete  the  three  tasks  collectively  increased  $22.8  million  over  what 
Customs  requested  in  the  first  plan.  Customs  funded  the  first  two  tasks  for  only 
6  months,  rather  than  18  months  as  specified  in  its  expenditure  plan.  (See  table 
on  next  page.) 
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Results:  Objective  3  (cont.) 


Task 

Summary  of  commitment  in  first  plan 

Actual  result 

Impact  on 
second  plan 

1.  Program  Establish  a  program  office  to  manage  the 
management  modernization  program  for  an  18-month  period. 


Work  was 
performed  for  6 
months. 


2.  Enterprise 
engineering 


Establish  and  maintain  the  systems  engineering  Work  was 
capability  for  implementing  ACE  for  an  18  month  performed  for  6 
period.  This  work  includes,  for  example,  updating  months, 
and  maintaining  Customs’  Enterprise  Architecture, 

Enterprise  Life  Cycle  Model,  and  Technical 
Reference  Model. 


$22.4  mihion  to 
fund  the 
remaining  12 
months  on  the 
first  two  task 
orders. 


3.  ACE  Develop  high-level  ACE  system  requirements,  and 

requirements  detailed  increment  1  requirements,  as  well  as  ACE 
and  planning  project  plan  outlining  the  transition  from  ACS  to 
ACE.  This  work  was  to  be  done  within  6  months. 


ACE  system  None, 

requirements, 
including  the 
detailed 

requirements  for 
ACE  increment  1, 
along  with 
project  plan, 
were  delivered 
within  6  months. 
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Results:  Objective  3  (cont.) 


According  to  Customs,  the  $25  million  request  in  the  first  expenditure  plan  was 
based  on  a  1999  estimate.  Customs  officials  acknowledged  that  derivation  of 
this  estimate  lacked  sufficient  rigor  because  it  was  not  based  on  detailed  work 
statements. 

Customs’  inability  to  meet  its  initial  commitments  casts  uncertainty  on  its 
ability  to  meet  future  commitments  and  thus  introduces  an  element  of  risk  to 
the  ACE  project. 
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Results:  Objective  3  (cont.) 


Observation  3:  Despite  progress,  Customs  does  not  yet  have  key 

management  controls  for  ACE. 

a.  Extension  of  the  enterprise  architecture  to  support  ACE 
engineering  tasks  is  not  complete. 

Enterprise  architectures  are  corporate  blueprints  to  guide  and  constrain 
operational  and  technological  change,  particularly  investments  in  large  system 
development  or  acquisition  efforts,  such  as  ACE.  To  effectively  guide  and 
constrain  these  efforts,  enterprise  architectures  should  be  updated  to  ensure 
that  they  reflect  changes  in  the  strategic  business  direction  of  the  enterprise. 
Further,  they  should  be  updated  in  a  way  that  ensures  consistency  and 
integration  across  the  full  scope  of  the  enterprise’s  business  areas;  for 
Customs,  these  areas  are  commercial,  enforcement,  and  administration.  Also, 
these  architectures  should  be  sufficiently  extended  to  recognize  system- 
specific  business  requirements  upon  which  later  design  decisions  will  be 
based.  Such  extensions  are  both  normal  and  a  necessary  part  of 
methodologically  sound  system  design  and  development.  To  minimize  risk, 
these  architectural  activities  should  be  completed  before  system  design  and 
development  begins. 
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Results:  Objective  3  (cont.) 


Customs  recognizes  the  need  to  ensure  that  its  ACE  system  architecture  is 
further  defined,  and  it  has  instituted  processes  for  providing  such  assurance. 
For  example,  it  is  having  eCP  extend  the  ACE  system  architecture  to  include 
logical  data  models  and  data  flows  and  expand  the  standards  for  Web  services. 
However,  Customs  has  yet  to  commit  to  work  on  its  enterprise  architecture 
that  eCP  officials  say  is  needed  to  ensure  that  Customs’  enterprise  architecture 
is  internally  consistent,  sufficiently  scoped,  and  adequately  extended. 

Instead,  Customs’  plans  to  provide  “just  enough”  enterprise  architecture  “just- 
in-time”  to  guide  and  constrain  the  design  of  future  ACE  increments.  According 
to  eCP  officials,  parallel  definition  of  the  enterprise  and  system  architectures  is 
not  typically  done.  The  reason  it  is  not  typically  done  is  because  it  risks  not 
having  sufficient  enterprise  context  available  to  effectively  design  the  system. 

We  believe  that  the  reason  for  Customs’  parallel  approach  is  the  priority  being 
given  to  meeting  the  ACE  schedule.  In  our  view,  such  a  schedule-driven 
approach  risks  producing  an  enterprise  architecture  extension  that  is  not 
sufficiently  complete  to  support  ACE  design  activities;  this,  in  turn,  could 
result  in  costly  system  rework  and  a  failure  to  deliver  system  capabilities  on 
time  and  within  budget. 
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Results:  Objective  3  (cont.) 


b.  Customs  does  not  have  an  effective  strategy  for  meeting  its 
modernization  human  capital  needs. 

Effective  human  capital  management  is  fundamental  to  any  organization  (or 
organizational  component),  such  as  the  CMO.  To  be  effective,  information 
technology  (IT)  human  capital  management  strategies  should  provide  for 

•  identifying  knowledge,  skills,  and  experience  requirements; 

•  continuously  assessing  existing  capabilities;  and 

•  developing  effective  strategies  and  implementing  plans  to  fill  any  gaps, 
including  developing  compensation  packages  to  attract  and  retain  qualified 
individuals. 

Since  the  CMO’s  establishment  in  June  2000,  its  approach  to  meeting  its  human 
capital  needs  has  been  to  identify  a  set  of  CMO  positions,  develop  position 
descriptions  for  each,  advertise  these  positions,  and  select  among  applicants. 

Using  this  approach,  the  CMO  currently  does  not  have  the  human  capital 
capacity  it  needs  to  effectively  manage  ACE.  (See  table  on  next  page.)  To 
compensate.  Customs’  approach  has  been  to  rely  on  support  contractors. 
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Results:  Objective  3  (cont.) 


Status  of  CMO  positions: 

Position  title 

Level 

FUled? 

Number  of  months  vacant 

Executive  Director 

SES 

Yes 

— 

Deputy  Director 

GS-15 

No 

2 

Requirements  Executive 

GS-15 

Yes 

— 

Requirements  Executive 

GS-15 

Yes 

— 

Executive  Support 

GS-14 

Yes 

— 

Executive  Support 

GS-7 

Yes 

— 

Requirements  Management  Lead 

GS-14 

No 

7 

Business  Management  Lead 

GS-14 

No 

5 

Commrmications  Lead 

GS-14 

No 

10 

Program  Control  Lead 

GS-14 

No 

0 

Systems  Engineering  Lead 

GS-13/14 

No 

9 

Program  Analyst 

GS-9/13 

No 

0 

Program  Analyst 

GS-9/13 

No 

5 
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Results:  Objective  3  (cont.) 


In  our  view,  the  CMO’s  lack  of  adequate  human  capital  can  be  traced  to  not 
having  defined  and  implemented  an  effective  human  capital  strategy.  Such  a 
strategy  would,  at  a  minimum,  completely  and  correctly  define  the  CMO’s 
human  capital  needs  (both  the  proper  skill  levels  and  the  commensurate 
compensation  packages  needed  to  attract  and  retain  them)  and  provide 
Customs’  full  commitment  (resources  and  management  attention)  for  pursuing 
short-term  and  long-term  actions  to  meet  them.  Unless  Customs  addresses  its 
human  capital  weaknesses,  it  increases  the  risk  of  not  delivering  ACE 
capabilities  on  time  and  within  budget. 
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Results:  Objective  3  (cont.) 


c.  Software  acquisition  management  processes  are  incomplete. 

Effective  IT  management  processes,  such  as  processes  for  acquiring  software, 
are  essential  to  software-intensive  system  acquisitions.  SEI,  recognized  for  its 
expertise  in  software  processes,  has  defined  the  tenets  of  effective  software 
acquisition.  The  SA-CMM  identifies,  among  other  things,  a  number  of  key 
process  areas  that  are  necessary  to  effectively  manage  software-intensive 
system  acquisition. 

As  discussed  earlier.  Customs  is  addressing  our  recommendations  aimed  at 
correcting  the  ACE  software  acquisition  management  weaknesses  that  we 
identified  in  our  February  1999  report,  and  Customs  officials  told  us  that  they 
plan  to  eventually  assess  their  SA-CMM  capabilities,  although  no  time  frame 
has  been  set. 


U.S  General  Accounting  Office,  Customs  Service  Modernization:  Serious  Management  and  Technical  Weaknesses  Must 
Be  Corrected,  GAO/AIMD-99-41,  Washington,  D.C.,  Feb.  26,  1999. 
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Results:  Objective  3  (cont.) 


Using  Customs’  reported  data  on  the  status  of  its  efforts  to  address  these 
weaknesses,  we  determined  that  one  of  the  SEI  key  process  areas  is  generally 
complete  (i.e.,  the  vast  majority  of  practices  in  a  key  process  area  are  in  place), 
five  key  process  areas  are  works  in  progress  (i.e.,  one  or  more  practices  in  a 
key  process  area  are  in  place),  and  two  key  process  areas  are  not  started  (i.e., 
no  practices  in  a  key  process  area  are  in  place). 
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Results:  Objective  3  (cont.) 


Status  at  Customs  on  SEI  key  process  areas: 

SEI  SA-CMM  key  process  area 

Generally 

complete 

Work  in  Not  started 
progress 

Software  acquisition  planning:  enstu’ing  that  reasonable  planning 
for  the  software  acquisition  is  conducted  and  that  all  elements  of 
the  project  are  included. 

•/ 

Solicitation:  ensming  that  award  is  made  to  the  contractor  most 
capable  of  satisfying  the  specified  requirements. 

•/ 

Requirements  development  and  management:  establishing  a 
common  and  imambiguous  definition  of  software  acquisition 
requirements  rmderstood  by  the  acquisition  team,  system  user,  and 
the  contractor. 

✓ 

Project  office  management:  managing  the  activities  of  the  project 
office  and  supporting  contractor(s)  to  ensure  a  timely,  efficient,  and 
effective  software  acquisition. 

•/ 

Contract  tracking  and  oversight:  ensuring  that  the  software 
activities  under  contract  are  being  performed  in  accordance  with 
contract  requirements,  and  that  products  and  services  will  satisfy 
contract  requirements. 

•/ 
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Results:  Objective  3  (cont.) 


SEI  SA-CMM  key  process  area 

Generally 

Work  in 

Not  started 

complete 

progress 

Evaluation:  determining  that  the  acquired  software  products  and 
services  satisfy  contract  requirements  prior  to  acceptance  and 
transition  to  support. 

✓ 

Transition  to  support:  providing  for  the  transition  of  the  software 
products  being  acquired  to  the  eventual  software  support 
organization. 

✓ 

Acquisition  risk  management:^  identifying  risks  as  early  as  possible, 
adjusting  acquisition  strategy  to  mitigate  those  risks,  and  developing 
and  implementing  a  risk  management  process  as  an  integral  part  of 
the  acquisition  process. 

•/ 

Acqtiisition  risk  management  is  an  SEI  SA-CMM  level  3  key  process  area. 
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Results:  Objective  3  (cont.) 


As  shown  in  the  table  above,  Customs  has  yet  to  complete  many  key  processes 
and,  in  particular,  the  evaluation  and  transition  to  support  key  process  areas 
have  not  been  started.  The  purpose  of  evaluation  is  to  determine  that  the 
acquired  software  products  and  services  satisfy  contract  requirements  before 
acceptance.  Evaluation  should  include  developing  and  managing  evaluation 
requirements  in  conjunction  with  developing  software  technical  requirements. 
Transition  to  support  is  to  provide  for  the  effective  and  efficient  “hand  off’  of 
the  acquired  software  products  to  the  support  organization  responsible  for 
software  maintenance  and  should  include  designating  a  group  that  is 
responsible  for  coordinating  transition  and  support  activities. 

Customs  officials  agree  that  its  software  acquisition  processes  are  incomplete 
and  told  us  that  Customs  delayed  addressing  the  evaluation  and  transition  to 
support  key  process  areas  because  resources  (i.e.,  staff)  were  needed  for 
higher  priority  activities.  In  the  absence  of  an  evaluation  capability.  Customs 
said  that  they  are  using  a  support  contractor  to  perform  evaluation  activities. 
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Results:  Objective  3  (cont.) 


Because  Customs  has  not  yet  established  effective  software  acquisition 
processes,  it  does  not  have  adequate  assurance  that  ACE  capabilities  will  be 
delivered  on  time  and  within  budget. 
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Results:  Objective  3  (cont.) 


d.  Expenditure  plan  cost  estimates  have  not  been  validated  by 
Customs. 


Producing  reliable  cost  estimates  for  inclusion  in  ACE  expenditure  plans  is 
essential.  Without  such  estimates,  Customs  cannot  provide  congressional 
committees  with  the  information  needed  to  exercise  important  ACE  oversight. 
Ensuring  that  cost  estimates  are  reliable  requires,  at  a  minimum,  that  Customs 
satisfy  itself  that  rigorous,  analytically  verifiable  methods  are  used  to  derive 
the  estimates.  To  this  end,  SEI  has  defined  models  for  (1)  establishing  an 
institutional  estimating  capability  and  (2)  validating  the  reliability  of  a  project- 
specific  estimate." 

As  discussed  earlier.  Customs’  cost  estimates  for  the  initial  ACE  tasks  in  its 
first  expenditure  plan  were  not  reliable  because  Customs  did  not  use  a 
rigorous  approach  in  deriving  them. 


Software  Engineering  Institute,  Checklists  and  Criteria  for  Evaluating  the  Cost  and  Schedule  Estimating  Capabilities 
of  Software  Organizations  fCMU/SEI-95-SR-005J  and  A  Manager’s  Checklist  for  Validating  Software  Cost  and  Schedule 
Estimates  rCMU/SEI-95-SR-004). 
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Results:  Objective  3  (cont.) 


For  its  second  expenditure  plan,  Customs  did  not  use  any  method  to  derive 
contract-related  costs.  Instead,  it  simply  passed  along  estimates  developed  by 
the  ACE  contractor,  except  for  Customs’  internal  (CMO)  costs  and  the 
management  reserve.  According  to  Customs,  it  plans  to  use  an  independent 
contractor  to  validate  cost  estimates  after  it  submits  future  expenditure  plans 
and  to  use  this  information  in  negotiating  task  orders. 

For  the  management  reserve  of  $35.5  million  requested  in  the  second 
expenditure  plan.  Customs  used  a  contractor  to  derive  this  estimate.  This 
estimate  represents  about  21  percent  of  the  total  funds  requested  (excluding 
the  reserve)  in  the  expenditure  plan.  We  found  that  this  reserve  estimate 
lacked  analytical  justification  and  relied  extensively  on  the  unsubstantiated 
assumptions  and  subjective  judgments  of  a  few  individuals.  In  fact,  we  could 
have  used  the  same  methodology  used  by  the  contractor,  made  our  own 
equally  valid  assumptions  and  judgments,  and  produced  estimates  either 
considerably  higher  or  lower. 
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Results:  Objective  3  (cont.) 


Moreover,  available  information  from  studies  and  comparable  system 
undertakings,  such  as  the  Internal  Revenue  Service’s  Business  Systems 
Modernization  (BSM),  do  not  support  this  level  of  management  reserve.  For 
example,  the  BSM  management  reserve  is  10  percent,  and  a  study  covering 
about  500  Defense  contracts  showed  a  mean  management  reserve  of  about  4 
percent. 

We  conclude  that  the  management  reserve  cost  estimate  appears  overstated, 
and  the  reliability  of  the  other  estimates  in  the  second  expenditure  plan  is 
uncertain.  Without  reliable  estimates.  Customs  increases  the  risk  that  it  will 
repeat  its  first  expenditure  plan  performance,  and  that  it  will  not  meet 
expenditure  plan  commitments. 
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Results:  Objective  3  (cont.) 


Observation  4:  Customs  has  accelerated  its  ACE  acquisition  plans. 

When  a  system  is  incrementally  acquired,  it  is  important  to  carefully  manage 
the  extent  to  which  design,  development,  testing,  and  implementation  of  the 
increments  overlap  (i.e.,  occur  concurrently).  The  goal  is  to  ensure  that  a  given 
increment  has  been  sufficiently  matured  (so  that  it  is  free  of  material  defects 
and  performs  as  intended)  before  later,  dependent  increments  are  built. 

While  some  degree  of  increment  concurrency  is  appropriate,  extensive  overlap 
of  increments’  detailed  design,  development,  and  testing  activities  is  generally 
not  desirable  because  it  increases  the  risk  of  overall  system  rework.  Reasons 
for  this  include,  carrying  over  unresolved  defects  into  later  increments; 
maintaining  control  over  a  proliferating  number  of  system  versions;  stretching 
critical  resources  (such  as  people,  development  and  testing  laboratories);  and 
introducing  incremental  change  into  the  organization  with  such  frequency  that 
it  exhausts  the  organization’s  capacity  to  absorb  the  change. 


Page  50  of  60 


Page  55 


GAO-02-545  Customs  Service  Modernization 


Appendix  I 

Customs’  Second  Automated  Commercial 
Environment  (ACE)  Expenditure  Plan 


i 


GAO 

Accountability  *  Integrity  *  Reliability 


Results:  Objective  3  (cont.) 


As  previously  stated,  Customs  has  committed  to  implementing  our  prior 
recommendation  to  incrementally  acquire  and  invest  in  ACE.  Until  recently, 
Customs’  incremental  strategy  provided  for  acquiring  and  deploying  ACE  in 
four  major  increments  spanning  5  years,  with  the  first  three  of  the  four 
increments  consisting  of  two  subincrements  or  releases.  Even  under  this  5-year 
strategy.  Customs’  plans  provided  for  some  overlap  among  increments.  For 
example,  design  activities  in  one  increment  were  generally  occurring 
concurrently  with  development  activities  in  later  increments.  (See  figure  on 
next  page.) 

In  January  2002,  Customs  decided  to  accelerate  its  plans  for  acquiring  and 
deploying  ACE,  electing  to  deliver  ACE’s  four  increments  in  4  years  rather  than 
5  because,  according  to  the  Commissioner  of  Customs,  ACE  is  important  to  our 
country’s  security  and  the  future  of  trade  facilitation.  While  Customs  has  yet  to 
develop  a  program  schedule  detailing  the  extent  to  which  this  compressed 
schedule  will  increase  concurrency  among  system  increment  design, 
development,  and  testing.  Customs  and  its  integration  contractor  acknowledge 
that  increased  concurrency  among  increments  will  occur. 
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Results:  Objective  3  (cont.) 


The  contractor  has  also  acknowledged  that  this  increased  concurrency  will 

exacerbate  preexisting  risks,  such  as 

•  the  possibility  that  an  increment’s  requirements  will  not  be  fully  understood 
before  design  and  development  is  to  begin; 

•  the  possibility  that  the  delivery  of  COTS  and  hardware  products  will  not  be 
timely;  and 

•  the  possibility  that  changes  will  be  required  to  Customs’  legacy  systems  with 
which  ACE  must  interface. 


The  contractor  has  also  stated  that  the  compressed  schedule  introduced  new 
risks,  such  as 

•  funding  requirements  will  be  so  high  that  budget  approval  is  unlikely; 

•  staffing  requirements  will  be  so  high  that  physical  housing  of  all  personnel 
will  be  difficult;  and 

•  the  trade  community  will  not  have  sufficient  lead  time  to  make  changes 
necessary  to  support  ACE. 
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Results:  Objective  3  (cont.) 


Combined  with  the  other  risks  that  we  cited  earlier  (i.e.,  unresolved  defects 
carrying  over  to  later  increments,  version  control,  overtaxing  core  resources, 
and  capacity  for  change  absorption),  the  chances  of  not  delivering  ACE 
capabilities  on  time  and  within  budget  are  increased  appreciably. 
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Conclusions 


Customs’  second  ACE  expenditure  plan  satisfies  the  legislative  conditions 
and  is  consistent  with  our  open  recommendations.  Nevertheless,  ACE 
introduces  a  host  of  unavoidable  risks,  including  the  complexity  of  the 
work,  the  dependence  of  successful  implementation  on  the  interaction  of 
a  large  and  diverse  set  of  stakeholders,  and  these  stakeholders’  ability  to 
absorb  the  considerable  operational  changes  to  be  introduced  through 
ACE. 


One  indicator  of  Customs’  ability  to  successfully  address  these  risks  and 
deliver  ACE  capabilities  on  time  and  within  budget  is  its  past 
performance.  Customs  did  not  fulfill  important  commitments  that  it  made 
in  its  first  expenditure  plan  because  the  plan’s  estimated  costs  for  certain 
tasks  were  not  reliable.  Because  of  this  track  record,  the  risks  in  moving 
forward  on  ACE  are  amplified. 
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Conclusions  (cont.) 


Another  indicator  of  Customs’  ability  to  successfully  acquire  and 
implement  ACE  is  the  effectiveness  of  the  management  controls  it  has  in 
place  to  govern  its  system  acquisition  and  implementation  activities.  For 
various  reasons  cited  by  Customs  officials,  including  limited  resources 
and  schedule  demands,  Customs  has  yet  to  establish  several  important 
management  controls,  such  as  an  updated  enterprise  architecture, 
adequate  human  capital  capacity,  complete  software  acquisition 
management  processes,  and  reliable  cost  estimating.  The  absence  of 
these  controls  further  compounds  the  risks  facing  Customs  on  ACE. 


Notwithstanding  these  risks.  Customs  has  recently  decided  to  develop 
and  deploy  the  system  faster.  This  decision  exacerbates  an  already  risky 
undertaking  by  increasing  the  difficulty  of  the  work  without  first 
providing  for  a  commensurate  increase  in  the  capabilities  needed  to 
perform  the  work. 
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Recommendations  for  Executive  Action 


To  address  the  risks  discussed  in  this  briefing,  we  recommend  that  the 
Customs  Service  Commissioner  direct  the  Chief  Information  Officer,  as  the 
designated  Modernization  Executive,  to  take  the  following  actions: 

•  Before  building  each  ACE  release  (i.e.,  beginning  detailed  design  and 
development),  certify  to  Customs’  House  and  Senate  appropriation 
subcommittees  that  the  enterprise  architecture  has  been  sufficiently 
extended  to  provide  the  requisite  enterprise  design  content  and  has  been 
updated  to  ensure  consistency  and  integration  across  business  areas. 

•  Immediately  develop  and  implement  a  CMO  human  capital  management 
strategy  that  provides  both  near-term  and  long-term  solutions  to  the  CMO’s 
human  capital  capacity  limitations,  including  defining  the  office’s  skill  and 
capability  needs  in  terms  that  will  allow  Customs  to  attract  qualified 
individuals  and  that  will  provide  sufficient  rewards  and  training,  linked  to 
performance,  that  promote  their  retention. 
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Recommendations  for  Executive  Action 


•  Develop  and  implement  process  controls  for  the  SEI  SA-CMM  level  2  key 
process  areas  and  the  level  3  acquisition  risk  management  key  process  area. 

•  By  September  30,  2002,  conduct  and  report  to  Customs’  House  and  Senate 
appropriation  subcommittees  on  the  results  of  an  internal  assessment  of  the 
CMO’s  maturity  against  the  SEI  SA-CMM  level  2  key  process  areas,  and  the 
level  3  acquisition  risk  management  key  process  area. 

•  Develop  and  implement  a  rigorous  and  analytically  verifiable  cost  estimating 
program  that  embodies  the  tenets  of  effective  estimating  as  defined  in  SEI’s 
institutional  and  project-specific  estimating  models. 

•  Limit  future  expenditure  plan  requests  for  management  reserve  funds  to  10 
percent  of  the  total  funds  requested  for  the  program  or  adequately  justify  any 
management  reserve  requests  in  excess  of  10  percent. 


Page  58  of  60 


Page  63 


GAO-02-545  Customs  Service  Modernization 


Appendix  I 

Customs’  Second  Automated  Commercial 
Environment  (ACE)  Expenditure  Plan 


i 


GAO 

Accountability  *  Integrity  *  Reliability 


Recommendations  for  Executive  Action 


•  Address  the  risks  associated  with  the  accelerated  ACE  acquisition  strategy, 
including  (1)  immediately  analyzing  the  risks  associated  with  the  degree  of 
design,  development,  and  testing  concurrency  across  ACE  increments  that  is 
inherent  in  Customs’  4-year,  schedule-driven  acquisition  strategy;  (2) 
reconsidering  the  merits  of  this  accelerated  strategy;  and  (3)  within  90  days 
of  the  date  of  this  briefing,  reporting  to  Customs’  House  and  Senate 
appropriations  subcommittees  on  the  agency’s  strategy  going  forward  and  its 
plans  for  mitigating  the  inherent  risks  associated  with  this  strategy. 


Page  59  of  60 


Page  64 


GAO-02-545  Customs  Service  Modernization 


Appendix  I 

Customs’  Second  Automated  Commercial 
Environment  (ACE)  Expenditure  Plan 


i 


GAO 

Accountability  *  Integrity  *  Reliability 


Agency  Comments 


In  commenting  on  a  draft  of  this  briefing,  the  Customs  Modernization  Office 
Executive  Director  agreed  with  our  findings,  conclusions,  and 
recommendations. 
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U.S.  Customs  Service 

Memorandum 


DATE:  April  24,  2002 
FILE:  AUD-1-OP  MD 


MEMORANDUM  FOR  RANDOLPH  C.  HITE 

U.S.  GENERAL  ACCOUNTING  OFFICE 

FROM:  Director 

Office  of  Planning 

SUBJECT:  Draft  Audit  Report  on  the  United  States  Customs 

Service’s  Second  Automated  Commercial 
Environment  (ACE)  Expenditure  Plan 

Thank  you  for  providing  us  with  a  copy  of  your  draft  report  entitled 
“Customs  Service  Modernization:  Management  improvements  Needed  on 
High-Risk  Automated  Commercial  Environment  Project”  and  the 
opportunity  to  discuss  the  issues  in  this  report. 

We  agree  with  GAO’s  overall  observations  that  the  Customs 
Modernization  program  is  large,  complex  and  important,  and  thus 
represents  considerabie  risk.  We  have  taken,  and  wiil  continue  to  take 
prudent  steps  to  address  the  risks  associated  with  the  Modernization 
program.  Key  actions  are  highlighted  below: 

•  While  we  believe  that  the  Customs  Modernization  Office  (CMO)  was 
appropriately  staffed  and  structured  for  the  early  phases  of  the 
program,  our  early  experiences  also  led  to  recognition  that  a  new 
structure  and  additional  resources  were  required  to  oversee 
application  development  and  deployment.  A  new,  expanded 
organizational  structure  has  been  approved  doubling  the  size  of  the 
CMO.  These  positions  are  being  filled  now. 

•  We  have  taken  a  number  of  steps  to  improve  our  cost  estimating 
capability  including  use  of  an  independent  government  cost  estimator 

Tradition  and  implementation  of  software  tools  that  allow  for  a  reliable  cost 

estimation  process.  These  efforts  will  be  reflected  in  future  Expenditure 
*  Plans. 


s 


E  RV  I  C  E 
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Although  the  original  cost  estimate  differed  from  the  actual  cost,  this  is 
not  representative  of  Customs  current  cost  estimating  capabilities.  The 
cost  estimates  from  the  first  ACE  expenditure  plan  pre-dated  the 
contract  award  and  did  not  include  task  order  changes  recommended 
by  eCP  and  approved  by  the  CMC.  Once  the  scope  changes  were 
approved,  Task  Orders  1-3  have  come  in  on  time,  on  budget,  and 
without  modification.  Customs  believes  that  now  that  the  task 
statements  are  being  developed  collaboratively  with  eCP,  and  with  the 
cost  estimation  improvements  that  CMO  has  implemented,  the  issue  of 
large  cost  variances  will  pass. 

•  Customs  and  eCP  recognize  the  risks  associated  with  pursuing  a 
4-year  application  rollout.  We  have  identified  the  inherent  risks  of  this 
4-year  approach,  analyzed  the  potential  impacts,  and  briefed  the 
Modernization  Executive  Steering  Committee.  The  CMO  and  eCP  are 
employing  risk  management  procedures  that  conform  to  Systems 
Engineering  Institute  standards  to  mitigate  risks  associated  with  the 
entire  Modernization  program,  including  the  first  ACE  release.  Prior  to 
receiving  authorization  to  proceed  in  each  of  the  subsequent  6 
releases,  we  commit  to  re-assessing  the  risk  environment  and 
developing  mitigation  strategies  to  meet  cost  and  quality  targets.  At 
any  point,  if  risks  cannot  be  credibly  mitigated  Customs  and  its 
Integration  Contractor  will  recommend  schedule  adjustments. 

•  Customs  has  tasked  eCP  to  update  and  extend  the  existing  enterprise 
architecture  in  concert  with  ACE  requirements,  planning  and 
engineering  work  to  provide  the  requisite  design  content,  and 
consistency  and  integration  across  the  full  scope  of  the  Customs 
business  areas.  We  will  do  this  in  an  incremental  and  logical  fashion 
to  minimize  risks  of  subsequent  re-work.  We  believe  we  can 
accomplish  this  objective  by  completing  the  basic  enterprise 
architecture  frameworks  prior  to  beginning  detailed  design  for  each 
increment.  Thus,  the  enterprise  architecture  frameworks  and  relevant 
derivative  artifacts  pertinent  to  the  specific  ACE  increment  content  will 
be  sufficiently  complete  prior  to  beginning  detailed  design  for  each 
increment. 
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•  The  CMO  is  proceeding  with  impiementation  of  software  acquisition 
controls  that  comply  with  the  Software  Engineering  Institute’s  Software 
Acquisition  Capability  Maturity  Model  and  is  75  percent  complete.  The 
CMO  continues  to  focus  on  the  most  critical  plans,  processes  and 
procedures  first  across  all  KPAs  as  we  proceed  through  the 
Modernization  effort. 

The  attachment  to  this  memorandum  details  the  specific  actions  that  are 
being  taken  to  respond  to  the  recommendations. 

If  you  have  any  questions  regarding  these  comments,  please  contact 
Ms.  Michele  Donahue  at  (202)  927-0957. 


Attachment 
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U.S.  CUSTOMS  SERVICE 

GAO  Review  of  Second  Automated  Commercial  Environment 
Expenditure  Pian 


Recommendation  1 :  Before  building  each  ACE  release  (i.e.,  beginning 

detailed  design  and  development),  certify  to  Customs’  House  and  Senate 
appropriation  subcommittees  that  the  enterprise  architecture  has  been 
sufficiently  extended  to  provide  the  requisite  enterprise  design  content  and  has 
been  updated  to  ensure  consistency  and  integration  across  business  areas. 

Response; 

Customs  recognizes  and  supports  the  imperative  to  estabiish  and  incrementally 
enhance  the  enterprise  architecture  to  effectiveiy  guide  and  constrain  large 
system  development  or  acquisition  projects  such  as  ACE.  Customs  has  tasked 
eCP  to  update  and  extend  the  existing  enterprise  architecture  in  concert  with 
ACE  requirements,  planning  and  engineering  work  in  order  to  provide  the 
requisite  design  content,  and  consistency  and  integration  across  the  full  scope  of 
the  Customs  business  areas.  The  tasking  and  scheduies  are  specifically 
designed  to  ensure  the  necessary  enterprise  architecture  framework  is  in  place 
to  adequately  guide  and  constrain  each  release  of  the  ACE  system,  while  also 
reflecting  any  resulting  changes  in  the  strategic  business  direction  of  the 
enterprise. 

Customs  agrees  with  the  GAO  recommendation  to  certify  that  extensions  to  the 
enterprise  architecture  provide  sufficient  context  to  effectively  design  the  system 
prior  to  building  each  ACE  release  and  reflect  any  necessary  strategic  business 
updates  required  to  ensure  consistency  and  integration  across  all  business 
areas.  eCP  has  recommended  tailoring  of  Customs  System  Development  Life 
Cycle  (SDLC)  to  include  a  Preliminary  Design  Review  (PDR)  which  will  include 
an  enterprise  and  ACE  system  architecture  assessment  component.  This  review 
occurs  prior  to  beginning  detailed  design  and  development,  and  therefore  is  the 
proper  point  in  the  lifecycle  to  demonstrate  the  necessary  architectural 
framework  and  content  exists  to  proceed.  The  results  of  this  review  will  provide 
the  objective  evidence  necessary  to  provide  certification  of  the  enterprise 
architecture  to  the  House  and  Senate  appropriation  subcommittees. 

Customs  expects  to  demonstrate  compliance  with  this  recommendation  prior  to 
the  start  of  Task  Order  4  detailed  design  work  in  June  2002. 

Target  Date;  June  30,  2002 
Responsible  Executive;  Charles  Armstrong 
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Recommendation  2:  Immediately  develop  and  implement  a  CMO  human 

capital  management  strategy  that  provides  both  near-term  and  long-term 
solutions  to  the  CMO’s  human  capital  capacity  limitations,  including  defining  the 
office’s  skill  and  capability  needs  in  terms  that  will  allow  Customs  to  attract 
qualified  individuals  and  that  will  provide  sufficient  rewards  and  training,  linked  to 
performance,  that  promote  their  retention. 

Response: 

Customs  believes  it  did  apply  appropriate  program  management  resources  to  the 
initial  phases  of  the  Modernization  program.  However,  Customs  concurs  with 
GAO  that  additional  resources  are  required  to  manage  the  expanding  complexity 
of  the  project.  To  that  end,  the  CMO  has  defined  a  new  organizational  structure 
for  managing  the  developmental  phases  of  the  Modernization  program  that  more 
than  doubles  the  government  positions  from  1 1  to  23.  Knowledge  and  skills 
required  to  perform  each  role  have  already  been  identified.  Customs  is 
proceeding  to  staff  the  new  organizational  structure  now,  and  by  May  1 , 2002, 
the  CMO  will  have  filled  a  number  of  positions  with  full-time.  Customs- 
experienced  individuals  leading  the  efforts  in  executive  support,  program 
management,  workforce  transition,  integration,  ACE  requirements  development, 
and  ACE  Increments  1  and  2  implementation.  New  CMO  members  will  be  quickly 
oriented  to  the  Modernization  Program  and  provided  necessary  training  as 
defined  in  the  Modernization  Training  Plan  for  Fiscal  Year  2002.  Customs  will 
continue  an  aggressive  recruiting  program  both  within  and  external  to  Customs  to 
fully  staff  the  CMO.  Further,  the  CMO  will  work  with  appropriate  other  Customs 
Offices  to  develop  a  human  resources  strategy  by  September  30,  2002  that  will 
focus  on  defining  core  competencies  to  support  recruiting,  retention  and  training 
efforts;  tying  performance  and  reward  programs  to  Modernization  goals;  and 
developing  a  succession  plan  to  enable  Customs  to  effectively  adapt  to 
personnel  changes. 

Target  Date:  September  30,  2002 
Responsible  Executive:  Charles  Armstrong 
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Recommendation  3:  Develop  and  implement  process  controls  for  the  SEI 

SA-CMM  level  2  key  process  areas  and  the  level  3  acquisition  risk  management 
key  process  area. 

Response; 

The  CMO's  comprehensive  SA-CMM  process  development  effort  is  75  percent 
complete  and  spans  all  of  SEI’s  SA-CMM  Level  2  Key  Process  Areas  (KPAs) — 
including  the  Evaluation  and  Transition  to  Support  KPAs — and  to  the  Level  3 
KPA,  Acquisition  Risk  Management  (ARM).  As  originally  noted  in  the  CMO’s 
briefing  to  GAO  on  February  26, 2001 ,  the  CMO  continues  to  focus  on  the  most 
critical  plans,  processes  and  procedures  first  across  all  KPAs  as  w/e  proceed 
through  the  modernization  effort. 

As  the  processes  are  completed,  they  are  immediately  released  to  the 
organization  for  use  in  an  institutionalization  effort.  Process  institutionalization 
will  continue  after  the  last  plan,  process  and  procedure  is  approved  and  will  end 
when  all  SA-CMM  process  artifacts  are  in  systematic  use  within  the  CMO. 

The  CMO  anticipates  that  the  process  development  activities  will  culminate  in  an 
internal  self-assessment  that  will  be  completed  by  September  2002,  and  the 
CMO  will  use  the  September  internal  self-assessment  to  baseline  the  process 
institutionalization  activities  as  well.  The  capstone  of  the  institutionalization  effort 
will  be  a  SA-CMM  external  pre-assessment  at  the  end  of  December  2002.  The 
results  of  the  external  pre-assessment  will  be  used  to  schedule  the  full  SEI 
assessment  in  2003. 

Target  Date:  September  30,  2002 
Responsible  Executive;  Charles  Armstrong 
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Recommendation  4:  By  September  30,  2002,  conduct  and  report  to 

Customs’  House  and  Senate  appropriation  subcommittees  on  the  results  of  an 
internal  assessment  of  the  CMO’s  maturity  against  the  SEI SA-CMM  level  2  key 
process  areas,  and  the  level  3  acquisition  risk  management  key  process  area. 

Response: 

The  CMO’s  current  Process  Improvement  schedule  supports  the  GAO's 
recommendation  to  conduct  and  report  to  Customs’  House  and  Senate 
appropriation  subcommittees  on  the  results  of  an  internal  assessment  of  the 
CMO’s  maturity  against  SA-CMM  Level  2  KPAs  and  the  Level  3  acquisition  risk 
management  KPA.  Customs  will  report  the  results  of  its  internal  self-assessment 
by  September  30,  2002. 

Target  Date:  September  30,  2002 
Responsible  Executive:  Charles  Armstrong 


Recommendation  5:  Develop  and  implement  a  rigorous  and  analytically 

verifiable  cost  estimating  program  that  embodies  the  tenets  of  effective 
estimating  as  defined  in  SEI’s  institutional  and  project-specific  estimating  models. 

Response: 

Customs  agrees  with  the  GAO  recommendation  that  a  rigorous  and  analytically 
verifiable  cost-estimating  program  is  essential  to  the  success  of  ACE.  Customs 
has  taken  measures  that  have  proven  successful  in  providing  a  solid  foundation 
on  which  the  program  has  effectively  contracted  for  three  large  Task  Orders  with 
no  cost  over-runs. 

While  there  was  a  variance  from  the  original  cost  estimates  for  Task  Orders  1 
and  2,  that  delta  was  due  to  a  two-year  old  estimate,  and  a  significant  change  in 
scope.  To  remedy  the  situation.  Customs  employed  an  independent  contractor, 
MCRI  to  develop  Independent  Government  Cost  Estimates  (IGCEs)  in  line  with 
SEI’s  estimating  checklist.  When  the  IGCEs  were  delivered  and  the  subsequent 
contract  negotiations  took  place.  Customs  was  able  to  issue  Task  Orders  1-3  that 
have  come  in  on-time,  on-budget,  and  without  modification.  Using  the  IGCE 
method  to  provide  independent  validation  of  eCP  estimates.  Customs  has 
recently  contracted  for  two  more  Task  Orders  (4  and  5)  that  fell  well  within  the 
acceptable  range  of  the  initial  estimates  from  MCRI. 

Customs  is  always  exploring  ways  to  refine  its  cost  estimating  practices,  and  will 
implement  improvements  as  they  are  identified.  One  such  improvement  that  has 
already  been  implemented  is  the  use  of  Crystal  Ball®  risk  analysis  software  to 
calculate  levels  of  certainty  around  different  estimating  values.  This  software  tool 
was  used  in  formulating  the  Expenditure  Plan  that  is  currently  undergoing 
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Executive  Branch  review  prior  to  submission  to  Congress.  This  software  tooi, 
employing  elements  of  SEI’s  cost  estimation  checklist,  will  be  used  to  formulate 
all  subsequent  expenditure  plans,  elevating  the  estimating  rigor,  providing  a 
greater  level  of  confidence  in  the  estimates,  and  allowing  for  better  planning  for 
known  program  risks. 

Target  Date;  Customs  believes  it  has  taken  actions  responsive  to  this 
recommendation. 

Responsible  Executive:  Charles  Armstrong 

Recommendation  6:  Limit  future  expenditure  pian  requests  for 

management  reserve  funds  to  10  percent  of  the  total  funds  requested  for  the 
program  or  adequately  justify  any  management  reserve  requests  in  excess  of  10 
percent. 

Response: 

The  CMC  understands  GAO  concerns  regarding  management  reserve.  To 
better  define  risk  and  uncertainty,  CMO  is  using  Crystal  Ball®,  a  risk  analysis  tooi. 
Consequently,  Customs  will  have  adequate  justification  for  future  expenditure 
plan  requests  for  management  reserve  funds. 

Target  Date:  Customs  believes  it  has  taken  actions  responsive  to  this 
recommendation. 

Responsible  Executive;  Charles  Armstrong 

Recommendation  7:  Address  the  risks  associated  with  the  accelerated 

ACE  acquisition  strategy,  including  (1)  immediately  analyzing  the  risks 
associated  with  the  degree  of  design,  development,  and  testing  concurrency 
across  ACE  increments  that  is  inherent  in  Customs  4-year,  schedule-driven 
acquisition  strategy,  (2)  reconsidering  the  merits  of  this  accelerated  strategy,  and 
(3)  within  90  days  of  the  date  of  this  briefing,  reporting  to  Customs  House  and 
Senate  appropriations  subcommittees  on  the  Agency’s  strategy  going  forward 
and  its  plans  for  mitigating  the  inherent  risks  associated  with  this  strategy. 

Response; 

Customs  and  its  Integration  Contractor  have  already  identified  the  inherent  risks 
of  this  4-year  approach,  analyzed  the  potential  impacts,  and  briefed  the 
Modernization  Executive  Steering  Committee.  The  CMO  and  eCP  are  employing 
risk  management  procedures  that  conform  to  Systems  Engineering  Institute 
standards  to  mitigate  risks  associated  with  the  entire  Modernization  program, 
including  the  first  ACE  release.  Prior  to  receiving  authorization  to  proceed  in 
each  of  the  subsequent  6  releases,  the  risk  environment  will  be  re-assessed  and 
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mitigation  strategies  will  be  developed  to  meet  cost  and  quality  targets.  At  any 
point,  if  risks  cannot  be  credibly  mitigated,  Customs  and  its  Integration  Contractor 
will  recommend  schedule  adjustments.  As  suggested  by  the  GAO,  Customs  will 
also  provide  a  report  to  the  House  and  Senate  Appropriation  Committees  by 
April  30,  2002,  outlining  the  risks  identified  and  the  corresponding  mitigation 
plans.  Within  this  same  time  period.  Customs  will  also  reconsider  and  analyze 
the  merits  of  the  current  ACE  program  plan  and  include  the  findings  in  this  report. 

Target  Date:  April  30,  2002 
Responsible  Executive:  Charies  Armstrong 


(310235) 
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GAO’s  Mission 

The  General  Accounting  Office,  the  investigative  arm  of  Congress,  exists  to 
support  Congress  in  meeting  its  constitutional  responsibilities  and  to  help  improve 
the  performance  and  accoimtability  of  the  federal  government  for  the  American 
people.  GAO  examines  the  use  of  public  funds;  evaluates  federal  programs  and 
policies;  and  provides  analyses,  recommendations,  and  other  assistance  to  help 
Congress  make  informed  oversight,  policy,  and  funding  decisions.  GAO’s 
commitment  to  good  government  is  reflected  in  its  core  values  of  accountability, 
integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no  cost  is 
through  the  Internet.  GAO’s  Web  site  (www.gao.gov)  contains  abstracts  and  full- 
text  files  of  current  reports  and  testimony  and  an  expanding  archive  of  older 
products.  The  Web  site  features  a  search  engine  to  help  you  locate  documents 
using  key  words  and  phrases.  You  can  print  these  documents  in  their  entirety, 
including  charts  and  other  graphics. 

Each  day,  GAO  issues  a  list  of  newly  released  reports,  testimony,  and 
correspondence.  GAO  posts  this  list,  known  as  “Today’s  Reports,”  on  its  Web  site 
daily.  The  list  contains  links  to  the  full-text  document  files.  To  have  GAO  e-mail  this 
list  to  you  every  afternoon,  go  to  www.gao.gov  and  select  “Subscribe  to  daily 
E-mail  alert  for  newly  released  products”  under  the  GAO  Reports  heading. 

Order  by  Mail  or  Phone 

The  first  copy  of  each  printed  report  is  free.  Additional  copies  are  $2  each.  A  check 
or  money  order  should  be  made  out  to  the  Superintendent  of  Documents.  GAO 
also  accepts  VISA  and  Mastercard.  Orders  for  100  or  more  copies  mailed  to  a  single 
address  are  discounted  25  percent.  Orders  should  be  sent  to: 

U.S.  General  Accounting  Office 

441  G  Street  NW,  Room  LM 

Washington,  D.C.  20548 

To  order  by  Phone:  Voice:  (202)  512-6000 

TDD:  (202)  512-2537 

Fax:  (202)  512-6061 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Web  site:  www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 

Public  Affairs 

Jeff  Nelligan,  managing  director,  NelliganJ@gao.gov  (202)  512-4800 
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